feat: add support for parallel kind processing with threads#840
Draft
bhearsum wants to merge 3 commits intotaskcluster:mainfrom
Draft
feat: add support for parallel kind processing with threads#840bhearsum wants to merge 3 commits intotaskcluster:mainfrom
bhearsum wants to merge 3 commits intotaskcluster:mainfrom
Conversation
Voluptuous contains thread-unsafe code; work around this by locking before calling into it.
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
from
1f3a474 to
2a48d4e
Compare
Even with 3.14 free-threaded python, this is still a bit slower than multiprocessing on Linux, but it will allow us to start experimenting with it more, and may allow users on macOS and Windows to immediately see a speed-up.
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
from
2a48d4e to
05e295a
Compare
Contributor
|
Msgspec does support freethreading in python: https://github.com/jcrist/msgspec/pull/877/files Initially thought we are planning to have both support for msgspec and voluptuous |
Contributor
Author
Oh, that's wonderful to hear about msgspec! Supporting both should be fine as far as I'm concerned for this work; it just means I'll need to rebase this after the schema work lands. |
|
msgspec support for 3.14 has been released https://github.com/jcrist/msgspec/releases/tag/0.20.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Even with 3.14 free-threaded python, this is still a bit slower than multiprocessing on Linux, but it will allow us to start experimenting with it more, and may allow users on macOS and Windows to immediately see a speed-up.
I did some basic testing of this on macOS in taskgraph; I haven't tested it in the context of Gecko yet outside of Linux. My intention is to turn it off by default for Gecko by setting
TASKGRAPH_SERIALwhen theforkmultiprocessing context is unavailable, and then get some more broad testing before it is enabled by default.I'll also probably wait until #752 in completed before landing this. This should either let me drop the
schema.pychanges altogether (ifmsgspecis threadsafe), or if it's not, there will be a lot of conflicts to deal with, and I don't want to interfere with that work.